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METHOD FOR IMPROVING DISK MIRRORING ERROR RECOVERY IN A COMPUTER SYSTEM 
INCLUDING AN ALTERNATE COmUNICATION PATH 



SPECIFICATION 

To all whom it may concern: 

Be it known that Richard Rollins, Michael 
Ohran, Randall C. Johnson, Scott Bonsteel, and 
Richard S. Ohran, citizens of the United States of 
America, have invented a new and useful invention 
entitled METHOD FOR IMPROVING ERROR RECOVERY 
PERFORMANCE IN A FAULT-TOLERANT COMPUTER SYSTEM of 
which the following comprises a complete 
specification . 
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METHOD FOR IMPROVING ERROR RECOVERY PERFORMANCE 
IN A FAULT-TOLERANT COMPUTER SYSTEM 

Microfiche Appendix , This specification includes a 
Microfiche Appendix which includes 1 page of 
microfiche and a total of 13 frames. The 
Microfiche Appendix includes computer source code 
illustrative of one preferred embodiment of the 
present invention* 

Background of the Invention 
Field of the Invention . This invention relates to 
fault-tolercuit computer systems, and in particular 
to the methods used to recover from a computer 
f ailxire in a system with redundant computers each 
with its own mass storage sy8tem(s). 
Descrip-tion of Wf^lat-giri An-, it is often desirable 
to provide continuous operation of computer 
systems, particularly file servers \rtiich support a 
number of user workstations or personal computers 
on a network. To achieve this continuous 
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operation, it is necessary for the computer system 
to be tolerant of software and hardware problems or 
faults. This is generally done by having redundant 
computers and redundant mass storage systems, such 
that a backup computer or disk drive is immediately 
available to take over in the event of a fault. 

A number of techniques for implementing a 
fault- tolerant coit^Juter system are described in 
Major et al.. United States Patent 5,157,663, which 
is hereby incorporated by reference in its 
entirety, and Major's cited references. In 
particular, the invention of Major provides a' 
replicated network file server capable of 
recovering from the failure of either the computer 
or the mass storage system of one of the two file 
servers. It has been used by Novell to implement 
its SFT-III fault-tolerant file server product. 

Figure 1 illustrates the hardware 
configuration for a fault-tolerant computer system 
100, such as described in Major. There are two 
server computer systems 110 and 120 connected to 
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network 101, from \^ich they receive requests from 
client computers. While we refer to computers 110 
and 120 as "server computer systems" or simply 
"servers" and show them in that role in the 
examples herein, this should not be regarded as 
limiting the present invention to computers used 
only as servers for other computer systems. 

Server computer system 110 has computer 
111 which includes a central processing unit and 
appropriate memory systems and other peripherals. 
Server computer system 120 has computer 121 which 
includes a central processing unit cind appropriate 
memory systems and other peripherals. Mass storage 
systems 112 and 113 are connected to computer 111, 
and mass storage systems 122 and 123 are connected 
to computer 121. Mass storage systems 112 and 123 
are optional devices for storing operating system 
routines and other data not associated with read 
and write requests received from network 101. 
Finally, there is an optional coimnunications link 
131 between computers 111 and 121. 
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The mass storage systems can be 
implemented using magnetic disk drives, optical 
discs, magnetic tape drives, or any other medium 
capable of handling the read and write requests of 
the particular computer system. 

An operating system or other control 
program runs on server computer systems 110 and 
120, executed by computers 111 and 121, 
respectively. This operating system heindles server 
requests received from network 101 and controls 
mass storage systems 112 and 113 on server 110, and 
mass storage systems 122 find 123 on server 12fl, as 
well as any other peripherals attached to computers 
111 and 121. 

While Figure 1 illustrates only two 
searver computer systems 110 and 120, because that 
is the most common (and lowest cost) configuration 
for a fault-tolerant con^uter system 100, 
configurations with more than two server computer 
systems are possible and do not depart from the 
spirit and scope of the present invention. 
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1 In normal operation, both server cojnputer 

2 system 110 and server computer system 120 handle 

3 each mass storage write request received firom 

4 network 101. Server computer system 110 writes the 

5 data from the network request to mass storage 

6 system 113, and server computer system 120 writes 

7 the data from the network request to mass storage 

8 system 122. This results in the data on loass 

9 storage system 122 being the mirror image of the 
.0 data on mass storage system 113 cmd the states of 
.1 server computer systems 110 and 120 are generally 
.2 consistent. In the following discussion, the 

.3 process of maintaining two or more identical copies 

-4 of information on separate mass storage systems is 

-5 referred to as "mirroring the information". 

-6 (For read operations, either server 

-7 computer system 110 or server computer system 120 

-8 can handle the request without involving the other 

-9 server, since a read operation does not change the 

10 state of the information stored on the mass storage 
:i systems . ) 

-6- 
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Although computer system 100 provides a 
substcintial degree of fault tolerance, ^en one of 
server computer systems 110 or 120 fails, the fault 
tolerance of the system is reduced. In the most 
common case of two server computer systems, as 
illustrated by Figure 1, the failure of one server 
computer system results in a system with no further 
tolerance to hcirdware faults or many software 
faults • 

In a f ault-tolercint computer system such 
as described above, it is necessary after a failed 
server computer system has been restored to bring 
the previously-failed computer systCTi into a state 
consistent with the server computer system that has 
continued operating. This requires writing all the 
changes made to the mass storage system of the non- 
failing server to the mass storage system of the 
previously-failed server so that the mass storage 
systems again mirror each other. Until that has 
been accontplished, the system is not fault tolerant 
even though the failed server has been restored. 
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If a server has been unavailable due to 
its failure for a period of time during \^ich there 
have been only a limited number of changes made to 
the mass storage sj^tem of the non-failing server, 
it is possible for the non-failing server to 
remember all the changes made (for example, by- 
keeping them in a list stored in its memory) and 
forward the changes to the previously-failed server 
when it has been restored to operation. The 
previously-failed server can then update its mass 
storage system with the chsmges and make it 
consistent with the non-failing server. This 
process typically does not cause excessive 
performance degradation to the non-failing server 
for any substantial period of time. 

However, if there have been more changes 
than can be conveniently remembered by the non- 
failing server, then the non-failing server must 
transfer all the information from its mass storage 
system to the previously-failed server for writing 
on its mass storage system in order to ensure that 
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the two servers are consistent. This is a very 
time consuming cuid resource-intensive operation, 
especially if the non-failing server must also 
handle server requests from the network \rtiile this 
transfer is taking place. For very large mass 
storage systems, as would be found on servers 
commonly in use today, and with a reasonably high 
network request load, it might be many hours before 
the mass storage systems are again consistent and 
the system is again fault tolerant. Additionally, 
the resource-intensiveness of the recovery 
operation can cause very stibstantial performance 
degradation of the non-failed server in processing 
network requests. 

Summary of the Invention 
It is an object of the present invention 
to provide tolerance to disk faults even though the 
computer of a server computer system has failed. 
This is achieved by electronically switching the 
laass storage system used for network requests from 
the failed server computer system to the non- 

-9- 
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failing server computer system. After the mass 
storage system from the failed server computer 
system has been connected to the non-failing 
server's computer, it is made consistent with the 
mass storage system of the non-failing server. 
This is typically a quick and simple operation* 
From that point on, the mass storage system from 
the failed server it is operated as a mirrored disk 
system, with each change being written by the non- 
failing server's computer to both the non-failing 
server's original mass storage system and to the 
mass storage system previously on the failed server 
and now connected to the non-f ailing server's- 
computer . 

While operating in this mode, the system 
will no longer be tolerant to processor failures if 
the non-failing server is the only remaining server 
(as would be the case in the common two-server 
configuration described above), but the system 
would be tolerant to failures of one of the mass 
storage systems. 

-10- 



wo 95/(M)906 



PCT/US94/07009 



It is a fur"ther object of the present 
invention to minimize the time the system is not 
fault tolercint by eliminating the need for time- 
consuming copying of the information stored on the - 
mass storage system of the non-failing server to 
the mass storage of the previously-failed server to 
make the two mass storage systems again consistent 
and permit mirroring of information again. 

This is also achieved by electronically 
switching the mass storage system from the failed 
server computer system to the non-failing server 
computer system. If this switch is accomplished 
after there have been only a small number of 
changes to the mass storage system of the non- 
failing server, the mass storage system from the 
failed server computer system can be quickly 
updated and made consistent, allowing mirroring to 
resume . 

Furthermore, since the mirroring of the 
invention keeps the information on the mass storage 
system from the failed server consistent while it 

-11- 
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is connected to the non-failing sever computer 
system, when the mass storage system is reconnected 
to the previously-failed server only those changes 
made between the time it was disconnected from the 
non-failed server and when it becomes available on 
the previously-failed server need to be made before 
it is again completely consistent and mirroring by 
the two servers (and full fault tolerance) resumes. 
This results in avoiding the substantial 
performance degradation experienced by the non- 
failing server during recovery using the prior art 
recovery method described above. As a result, the 
invention provides rapid recovery from a fault in 
the system. 

These and other features of the invention 
will be more readily xinderstood upon consideration 
of the attached drawings and of the following 
detailed description of those drawings and the 
presently preferred embodiments of the invention. 
Brief Description of the Drawings 



-12- 
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1 Figure 1 illustrates a prior art 

2 implementation of a f ault-tolerant computer system 

3 with two server computer syst^is. 

4 Figure 2 illustrates the fault-tolerant 

5 computer system of Figure 1, modified to permit the 

6 method of the invention by includdLng means for 

7 connecting a mass storage system to either server's 

8 computer • 

9 Figure 3 is a flow diagram illustrating 
10 the steps to be taken when a processor failure is 
u detected. 

12 Figure 4 is a flow diagram illustrating 

13 the steps to be taken when the previously-failed 
L4 processor becomes available. 

15 Detailed Description of the Invention 

L6 Referring to fault-tolerant cQn5)uter 

L7 system 200 of Figure 2, and comparing it to prior 

18 art fault- tolerant computer system 100 as 

L9 illustrated in Figure 1, we see that mass storage 

10 systems 113 and 122, \diich were used for storing 

2i the information read or written in response to 

-13- 
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requests from other computer systems on network 
101/ are now part of reconf igurable mass storage 
system 240. In particular , xaass storage system 113 
can be selectively connected by connection means 
241 to either computer 111 or computer 121 (or 
possibly both computers 111 and 121, although such 
dual connection is not necessary for the present 
invention), and mass storage system 122 can 
likewise be independently selectively connected to 
either computer 111 or computer 121 by connection 
means 241. The mass storage system 240 is 
reconfigurable because of the ability to select and 
change connections between mass storage devices and 
computers . 

While Figure 2 illustrates the most 
common dual server conf igiiration emticipated by the 
inventors / other configurations with more than two 
servers are within the scope of the present 
invention / and the extension of the techniques 
described below to other configurations will be 
obvious to one skilled in the art. 
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There are a number of ways such 
connection means 241 ccui be implemented, depending 
on the nature of the mass storage system interface 
to computers 111 or 121. Connection means 241 can 
be two independent two-channel switches, \^ich 
electronically connect all the interface signals 
from a mass storage system to two computers. Such 
two-channel switches may be a part of the mass 
storage system (as is conmion for mass storage 
systems intended for use with mainframe computers) 
or can be a separate unit. A disadvamtage of using 
two-channel switches is the large number of 
switching gates that are necessary if the number of 
data and control lines in the mass storage 
interface is large. That number increases rapidly 
when there are more than two server computer 
systems in fault-tolerant computer system 200. For 
example, a fault- tolerant computer system with 
three computers coxmected to three mass storage 
syst^os would require 2.25 times the number of 
switching gates as the system illustrated in Figure 

-15- 
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1 2. (The number of switching gates is proportional 

2 to the number of computers times the nimber of mass 

3 storage systems • ) The number of switching gates 

4 can be reduced by not connecting every mass storage 

5 system to every computer, although such a 

6 configuration would be less flexible in its 

7 reconfiguration cibility. 

8 Another implementation of connection 

9 mefiuis 241 is for both computer 111 and computer 121 
.0 to have interfaces to a common bus to which mass 

11 storage systems 113 and 122 are also connected. An 

12 example of such a bus is the small computer system 
-3 interface (SCSI) as used on many workstations and 
A personal computers. When a computer wishes to 

.5 access a mass storage system, the computer recjuests 

16 ownership of the bus through an appropriate bus 

L7 arbitration procedure, and when ownership is 

L8 granted, the computer performs the desired loass 

19 storage operation. A disadvantage of this 

10 implementation is that only one computer (the one 
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with current bus ownership) can access a mass 
storage system at a time. 

If it is desirable to use a standard SCSI 
bus as mecuis 241 for connecting mass storage 
systems 113 and 122 to computers 111 and 121, cmd 
to allow simultaneous access of the mass storage 
systems 113 and 122 by their respective server's 
computers, computers 111 and 121 can each have two 
SCSI interfaces, one connected to mass storage 
system 113 and one connected to mass storage system 
122. Mass storage system 113 will be on a SCSI bus 
connected to both computers 111 and 121, and mass 
storage system 122 will be on a second SCSI bus, 
also connected to both computers 111 and 121 • If 
computer 111 or computer 121 is not using a 
particular mass storage system, it will configure 
its SCSI interface to be inactive on that mass 
storage systems particular bus. 

In the preferred embodiment, a high-speed 
serial network between computers 111 and 121 and 
mass storage systems 113 and 122 forms connection 
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means 241. Each computer 111 contains an interface 
to the network, and requests to a mass storage 
system 113 or 122 are routed to the appropriate 
network interface serving the particular mass 
storage system. Although a bus-type network, such 
as an Ethernet, could be used, the network of the 
preferred embodiment has network nodes at each 
computer and at each mass storage system. Each 
node can be connected to up to four other network 
nodes. A message is routed by each network node to 
a next network node closer to the message's final 
destination. 

For the fault-tolerant computer system 
configuration of Figure 2, one network connection 
from the node at con^uter 111 is connected to the 
node for mass storage system 113, and another 
network connection from the node at computer 111 is 
connected to the node for mass storage system 122. 
Similar connections are used for computer 121. 
Mass storage system 113 's node is connected 
directly to the nodes for computers 111 and 121, 

-18- 
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1 and mass storage system 122 's node is similarly 

2 connected (but with different links) to computers 

3 111 cind 121. Routing of messages is trivial, since 

4 there is only one link between each computer and 

5 each mass storage system. 

6 The particular connecting means 241 used 

7 to connect computers 111 and 121 to mass storage 

8 systems 113 and 122 is not critical to the method 

9 of the present invention, so long as it provides 
LO for the rapid switching of a mass storage system 
Ll from one computer to another without affecting the 
L2 operation of the computers. Any such means for 

13 connecting a mass storage system to two or more 

14 computers is usable by the method of the present 

15 invention . 

16 The method of the present invention is 

17 divided into two portions, a first portion for 

18 reacting to a processor failure and a second 

19 portion for recovering from a processor failure. 

10 The first portion of the method of the present 

11 invention is illustrated by Figure 3, which is a 

-19- 
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flow diagram illustrating the steps to be taken 
when a processor failure is detected. The 
description of the method provided below should be 
read in light of Figure 2. For purposes of 
illustration, it will be assumed that coimection 
means 241 initially connects mass storage system 
113 to computer 111 and mass storage system 122 to 
computer 121, providing an equivalent to the 
configuration illustrated in Figure 1 although the 
connection means 241 of Figure 2 facilitates this 
equivalent configuration. Information mirroring as 
described above is being performed by computers 111 
and 122. It is also assumed that computer 121 has 
experienced a fault, causing server computer system 
120 to fail. 

The method starts in step 301, with each 
computer 111 and 122 waiting to detect a failxire of 
another sejrver's computer 111 and 122. Such 
failure can be detected by probing the status of 
the other server's coii5)uter by a means appropriate 
to the particular operating system being used and 



WO^/00906 



PCTAJS94/a7009 



the communications methods between the servers. In 
the case of Novell's SFT-III^ the method will be 
running as a NetWare Loadable Module, or NLM, and 
be capable of communicating directly with the' 
operating system by means of requests. The NLM 
will make a null request to the SFT-III process. 
This null request will be such that it will never 
normally run to completion, but will remain in the 
SFT-III process queue. (It will recpiire minimal 
resources ^i^ile it remains in the process queue.) 
In the event of a failure of server conqputer system 
121, SFT-III running on server computer system 111 
will indicate the failure of the null request to 
the NLM of the method, indicating the failure of 
server 121. Because a processor failure has been 
detected, the method depicted in Figure 3 proceeds 
to step 302. 

In step 302, detection of the failure of 
server 121 causes the discontinuation of mirroring 
information on the failed server 121. This 
discontinuation can either be done automatically by 

-21- 
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1 the operating system upon its detection of the 

2 failure of server 121, or by the particular 

3 implementation of the preferred embodiment of the 
A method of the present invention. In the case- of 

5 SPT-III, the discontinuation of mirroring on server 

6 121 is performed by the SPT-III operating system. 

7 Step 303 of the method is performed next. 

8 In step 303, SFT-III remembers all data 

9 not mirrored on server 121 following its failure as 
.0 long as the amount of data to be remembered does 

-1 not exceed the capacity of the system resotirce 

-2 remembering the data. If the particular operating 

.3 system does not remember non -mirrored data, step 

.4 303 would have to be performed by the particular 

.5 implementation of the method of the present 

.6 invention. The step of remembering all non- 

.7 mirrored data could be performed by any technique 

.8 known to persons skilled in the art. 

-9 Next, step 304 of the method sets 

:0 connection means 241 to discoimect mass storage 

-1 system 122 from con5>uter 121 of failed server 120, 

-22- 
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and to connect it to computer 111 of non-failing 
server 110. At this point, the method can quickly 
test mass storage system 122 to determine if it is 
the cause of the failure of seinrer 120. If it is, - 
there is no fault-tolerance recovery possible using 
the method, and laass storage system 122 can be 
disconnected from computer 111 at connection means 
241. If mass storage system 122 is not the cause 
of server 120 's failure, then the cause must be 
computer 121, and the method can continue to 
achieve limited fault tolerance in the presence of 
the computer 121 's failure. 

Step 305 commands the operating system of 
server 110 to scan for new mass storage systems, 
causing the operating system to determine that mass 
storage system 122 is now connected to computer 
111, along with mass storage system 113. SFT-III 
will detect through information on inass storage 
systems 113 and 122 that they contain similar 
information, but that mass storage system 122 is 
not consistent with mass storage systrai 113. In 

-23- 
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step 306, SFT-III will update mass storage system 
122 using the information remembered at step 303 
and, after the two mass storage systems are 
consistent (i.e., contain identical mirrored copies 
of the stored information), step 307 will begin 
mirroring all information on both mass storage 
systems 113 and 122 and resume normal operation of 
the system. If an operating system different than 
SFT-III does not provide this automatic update for 
consistency and mirroring, the implementation of 
the method will have to provide an equivalent 
service . 

Note that when SFT-III is used, the only 
steps of the method that must be performed by the 
NETWARE loadable module are; (1) detecting the 
failure of server 120 (step 301), (2) setting 
communications means 241 to disconnect mass storage 
system 122 from conqputer 121 and connecting it to 
computer 111 (step 304), (3) determining if mass 
storage system 122 was the cause of the failure of 
server 120 (also part of step (304), and (4) 

-24- 
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commanding SFT-III to scan for mass storage systems 
so that it finds the newly-connected mass storage 
system 122 (step 305). All the other steps are 
performed as part of the standard facilities of 
SFT-III. In other embodiments of the invention, 
responsibility for performing the steps of the 
method may be allocated differently. 

Figure 4 is a flow diagram illustrating 
the second portion of the invention - the steps to 
be taken when previously-failed server 120 becomes 
available again. Server 120 would typically become 
available after correction of the problem that 
caused its failure described above. Step 401 
determines that server 102 is available and the - 
method proceeds to step 402. In step 402, the 
method sets connection means 241 to disconnect mass 
storage system 122 from computer 111 after 
coimnanding SFT-III on server 110 to remove mass 
storage system 122 from its active mass storage 
systems. Due to the tinavailability of mass storage 
system 122 on server 110, data mirroring on seinrer 

-25- 
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110 will be stopped by SFT-III and it will begin 
remembering chemges to mass storage system 113 not 
made to mass storage system 122 to be used in 
making the storage systems consistent later. 

In step 403, mass storage system 122 is 
reconnected to computer 121, and in step 404, SFT- 
III on server 120 is commcinded to scan for the 
newly-connected mass storage system 122. This 
returns mass storage system 122 to the con^iuter 121 
to which it was originally connected prior to a 
server failure. When SFT-III on server 120 detects 
mass storage system 122, it communicates with 
server 110 over link 131. At this point, the 
operating systems on servers 110 and 120 work 
together to make mass storage system 122 again 
consistent with mass storage system 113 (i.e.-, by 
remembering interim changes to mass storage system 
113 and writing them to mass storage system 122), 
and when consistency is achieved, data mirroring on 
the two servers resumes. At this point, recovery 
from the server failure is complete. 
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In an SFT-III system, the only steps of 
the method that the NetWare Loadable Module must 
perform are: (1) detecting the availability of 
server 120 (step 401), (2) removing mass storage 
system 122 from the operating system on server 110 
(step 402), (3) disconnecting mass storage system 
122 from computer 111 and connecting it to computer 

121 by setting connection means 241 (step 403), and 
(4) commanding SFT-III on server 120 to scan for 
mass storage so that it locates mass storage system 

122 (step 404 )• The steps involved with making 
mass storage systems 113 and 122 consistent and 
reestablishing data mirroring (step 405) are 
performed as part of the standard facilities of 
SFT-III. In other embodiments of the invention, 
responsibility for the steps of the method may be 
allocated differently. 

Figure 2 illustrates optional mass 
storage systems 112 and 123 attached to computers 
111 and 121, respectively. While these two mass 
storage systems are not recpiired by the method of 
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the present invention^ they are useful during the 
restoration of a failed server. They provide 
storage for the operating system and other 
information needed by failed server 120 to begin 
operation before mass storage system 122 is 
switched from computer 111 to computer 121. Were 
mass storage system 123 not available, some means 
of having mass storage system 122 connected both to 
computer 121 {for initializing its operation 
following correction of its failure) and computer 
111 (for continued disk mirroring) would be 
necessary. Alternatively, if the initialization 
time of server 120 is short, mass storage system 
122 could be switched from computer 111 to computer 
121 at the start of server 120 's initialization, 
though this would result in more changes that must 
be remembered and made before data mirroring can 
begin again. 

It is to be understood that the above 
described embodiments are merely illustrative of 
numerous and varied other embodiments which may 
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1 constitute applications of the principles of the 

2 invention. Such other exabodiiaents may be readily 

3 devised by those skilled in the art without 

4 departing from the spirit or scope of this 

5 invention and it is our intent they be deemed 

6 within the scope of our invention. 

7 
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1 

2 Claims 

3 We claim: 

4 1. A method for rapid failure recovery and 

5 system restoration in a fault-tolercoit computer 

6 system, said computer system comprising: 

7 (A) a first server computer system, 

8 comprising a first computer executing an 

9 operating system; 

0 (B) a second server con^uter system, 

-1 comprising a second computer executing an 

.2 operating syst^; 

.3 (C) a first mass storage system connected to 

4 said first computer; 

.5 (^) ^ second mass storage system; and 

•6 (E) means for coimecting said second mass 

.7 storage system to said first con^uter and to 

.8 said second cozt^uter; ' 

.9 WHEREIN whenever said first computer writes 

:0 data to said first mass storage system, said second 
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computer writes a mirror copy of said data to said 

second mass storage system, 

the method comprising the steps of: 

(1) detecting a failure of said second 
conqputer; 

(2) discontinuing causing said writing of 
said mirror copy on said second mass storage 
system; 

(3) remembering data written to said first 
mass storage system but not written to said 
second mass storage system; 

(4) configuring said second mass storage 
system to record information from said first 
computer; 

(5) writing said remembered data to said 
second mass storage system; 

(6) \(rtienever new data is written to said 
first mass storage system , writing a mirror 
copy of said new data to said second mass 
storage system; 
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1 (7) detiectlng said second coinput:er's 

2 availability ; 

3 (8) reconfiguring said second mass storage 

4 system to recoxrd information from said second 

5 computer; 

6 (9) reestablishing data mirroring such that 

7 whenever said first computer writes data to 

8 said first mass storage system, said second 

9 computer writes a mirror copy of said data on 

0 said second mass storage system. 

1 2. A method as in claim 1 \^erein step (1) is 

2 performed by said first computer. 

3 3. A method as in claim 2 wiierein step (2) is 

4 performed by said first computer. 

5 4 . A method as in claim 1 wherein step ( 3 ) is 

6 performed by said first computer. 

7 5. A method as in claim 4 \dierein step (5) is 

8 performed by said first computer. , 

9 6. A method as in claim 5 iirtierein step (6) is 
0 performed by said first computer. 
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1' 7. A method as in claim 1, \^erein said first 

2 mass storage system and said second mass storage 

3 system each con^rise at least one magnetic disk 

4 drive . 

5 8. A method as in claim 1/ \^erein said means 

6 £or connecting said second mass storage system 

7 comprises a serial network. 

8 9. A method as in claim 1 wherein said operating 

9 systems are the SFT-III operating system. 

0 10. A method as in claim 9 wherein steps (1), (4) 

1 and (5) are performed by a NETWARE loadable module. 

.2 

3 11. A method for rapid failtire recovery and 

4 system restoration in a fault-tolerant computer 

5 system r said computer system coiqprising: 
.6 (A) a first server computer system, 

.7 comprisdLng a first computer executing an 

.8 operating system; 

.9 (B) a second server computer system, 

:o comprising a second computer executing an 

.1' operating system; 
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1 (C) ^ first mass storage system connected to 

2 said first computer; 

3 (I^) ci second mass storage system; and 

4 (E) means for selectively connecting said 

5 second mass storage system to said first 

6 computer and to said second computer; 

7 WHEREIN in the absence of a fault said second 

8 mass storage system is connected to said second 

9 computer ; and 

0 WHEREIN \f^enever said first computer writes 

1 data to said first mass storage system said first 
.2 computer can also cause said second computer to 

3 write a mixrror copy of said data to said second 

4 mass storage system, 

5 the method of the invention comprising: 

.6 (1) on said first computer, detecting a 

.7 failure of said second computer; 

.8 (2) on said first computer, discontiniiing 

.9 causing said writing of said mirror copy on 

.0 said second mass storage system by said 

.1 second computer; 
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1 (3) on said first computer, remembering data 

2 written to said first mass storage system but 

3 not written to said second mass storage 

4 system; 

5 (4) on said first computer, setting said 

6 means for connecting said second mass storage 

7 system to connect said second mass storage 

8 system to said first computer; 

9 (5) on said first computer, commanding said 
-0 operating system of said first conqputer to 

<l seem for mass storage systems such that said 

-2 operating system of said first computer will 

.3 determine that both said first mass storage 

.4 system and said second mass storage system 

-5 are now connected to said first computer; 

16 (6) on said first computer, writing said 

17 remembered data to said second mass storage 

18 system; ' 

19 (7) on said first computer, whenever new data 
10 is written to said first mass storage system. 
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1 writing a mirror copy of said new data to 

2 said second mass storage system; 

3 (8) on said first computer, detecting said 

4 second computer's availability; 

5 _ (9) on said first computer, commanding said 

6 operating system of said first computer to 
. 7 remove said second mass storage system; 

8 (10) setting said means for connecting said 

9 second mass storage system to connect said 

10 second mass storage system to said second 

11 computer; 

12 (11) on said second computer, commanding 

13 said operating system of said second computer 

14 to scan for mass storage systems such that 

15 said operating system of said second computer 

16 will determine that said second mass storage 

17 system is now coxmected to said second 
L8 computer; 

L9 (12) reestablishing data mirroring such that 

zo v^enever said first computer writes data to 

21 said first mass storage syston said first 
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1 computer also causes said second computer to 

2 write a mirror copy of said data on said 

3 second mass storage system. 

4 12* A method as in claim 11, wherein said first 

5 mass storage system and said second mass storage 

6 system each comprise at least one magnetic disk 

7 drive. 

8 13. A method as in claim 12, wherein said means 

9 for connecting said second mass storage system 
10 comprises a serial network. 

11 

12 14. A method for rapid failure recovery in a 

L3 fault- tolerant computer system said computer 

14 system comprising: 

15 (A) a first server computer system, 

16 comprising a first cos^uter executing an 

17 operating system; 

18 (B) a second server con5>uter system, 

19 comprising a second computer; 

20 (C) a first mass storage system connected to 

21 said first computer; 
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1 (I^) ci second mass storage system; and 

2 (E) means for selectively connecting said 

3 second mass storage system to said first 

4 computer and to said second computer; 

5 WHEREIN in the absence of a fault said second 

6 mass storage system is connected to said second 

7 computer; and 

8 WHEREIN v^enever said first computer writes 

9 data to said first mass storage system said first 

10 computer ccui also cause said second computer to 

11 write a mirror copy of said data on said second 

12 mass storage system, 

13 the method of the invention comprising said first 

14 computer performing the steps of: 

15 (1) detecting a failure of said second 

16 computer; 

17 (2) discontinuing causing said writing of 

18 said mirror copy on said second mass storage 

19 system by said second computer; 
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1 


(3) remembering data written to said first 


2 


mass storage system but not written to said 


3 


second mass storage system; 


4 


(4) setting said means for connecting said 


5 


second mass storage system to connect said 


6 


second mass storage system to said first 


7 


computer; 


8 


(5) commanding said operating system of said 


9 


first con^uter to scan for mass storage 


10 


systems such that said operating system, of 


11 


said first computer will determine that both 


12 


said first mass storage system and said 


13 


second mass storage system are now connected 


14 


to said first computer; 


15 


(6) writing said remembered data to said 


16 


second mass storage system; 


17 


(7) whenever new data is written to said 


18 


first mass storage system, writing a mirror 


19 


copy of said new data to said second mass 


20 


storage s^^stem. 
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1 15. A method as in claim 14, wherein said first 

2 mass storage system and said second mass storage 

3 system each comprise at least one magnetic disk 

4 drive . 

5 16. A method as in claim 15, wherein said means 

6 for connecting said second mass storage system 
1 comprises a serial network. 

8 

9 17. A method for system restoration in a fault- 

10 tolerant computer system, said computer system 

u. comprising: 

L2 (A) a first server computer system, 

L3 comprising a first computer executing em 

L4 operating systCTi; 

15 (B) a second server computer system, 

16 comprising a second computer executing an 

17 operating systCTi; 

L8 (C) ^ first mass storage system coimected to 

L9 said first computer; 

20 (D) a second mass storage system; and 
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1* (E) means for connecting said second mass 

2 storage system to said first computer and to 

3 said second computer; 

4 WHEREIN said second computer is initially 

5 unavailable for use, and 

6 WHEREIN said second mass storage system is 

7 initially connected to said first computer, the 

8 method comprising: 

9 (1) on said first computer, detecting said 

0 second computer's availcibility; 

1 (2) on said first computer, conmicmding said 
.2 operating system of sa±d first computer to 

3 remove said second mass storage system; 

4 (3) setting said means for connecting said 
3 second mass storage system to connect said 
.6 second mass storage system to said second 
-7 computer; 

.8 (4) on said second computer, commanding said 

-9 operating system of said second computer to 

.0 sc£Ln for mass storage systems such that said 

.1" operating system of said second computer will 
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1 determine that said second mass storage 

2 system is now connected to said second 

3 computer; 

4 (5) reestablishing data mirroring such that 

5 whenever said first computer writes data to 

6 said first mass storage system said first 

7 computer also causes said second computer to 

8 write a mirror copy of said data on said 

9 second mass storage system. 

.0 18. A method as in claim 17 ^ wherein said first 

.1 xaasB storage system and said second mass storage 

.2 system each comprise at least one magnetic disk 

.3 drive • 

A 19. A method as in claim 18, wherein said means 

,5 for connecting said second mass storage system 

-6 comprises a serial network. . 

-7 20. A method as in claim 17 wherein said 

18 operating system is the SFT-III operating system. 

.9 21. A method as in claim 20 wherein steps (1), 

>0 (4) and (5) are performed by a NETWARE loadable 

11 module . 

«42" 



wo 95/00906 



PCT/US94/07009 



1 

2 22. A method for rapid failure recovery in a 

3 fault- tolerant computer system, said computer 

4 system conqprising: 

5 (A) a first seirver computer system, 

6 comprising a first computer executing an 

7 operating system; 

8 (B) a second server computer system, 

9 comprising a second computer executing an 
.0 operating system; 

.1 (C) a first mass storage system coxmect^ to 

.2 said first computer; 

.3 (D) ^. second mass storage system; and 

.A (E) means for connecting said second mass 

.5 storage system to said first computer and to 

.6 said second computer; 

.7 WHEREIN \^enever said first computer writes 

.8 data to said first iDass storage system, said second 

-9 computer writes a mirror copy of said data to said 

10 second mass storage system, 

:i the method comprising the steps of: 
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1 (1) detecting a failure of said second 

2 computer; 

3 (2) discontinuing causing said writing of 

4 said mirror copy on said second mass storage 

5 system; 

6 (3) remembering data written to said first 

7 mass storage system but not written to said 

8 second mass storage system; 

9 (4) configuring said second mass storage 

.0 system to record information from said first 

.1 computer; 

-2 (5) writing said remembered data to said 

.3 second mass storage system; and 

.4 ( 6 ) , whenever new data is written to said 

-5 first mass storage system, writing a mirror 

.6 copy of said new data to said second xaass 

.7 storage system. 

.8 

.9 23. A method for system restoration in a fault- 

:o tolerant computer system, said computer syst^ 

:i comprising: 
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L (A) a first server computer system, 

2 comprising a first computer executing an 

3 operating system; 

4 (B) a second server computer system, 

5 comprising a second computer executing an 

6 operating system; 

7 (C) a first mass storage system connected to 

8 said first computer; 

9 (D) a second mass storage system; 

0 (E) means for connecting said second mass 

.1 storage sYBtem to said first computer and to 

.2 said second computer; 

3 ^THEREIN said second computer is Initially 

.4 unavaileJ^le for use; £md 

.5 WHEREIN said second mass storage system is 

.6 initially configured to record information from 

.7 said first con^uter, 

.8 the method con^rising the steps of: 

.9 (1) detecting said second computer's 

:o availability; 
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1 (2) reconfiguring said second mass storage 

2 system to record information from said second 

3 computer; 

4 (3) establishing data mirroring such that 

5 whenever said first computer writes data to 

6 said first mass storage system, said second 

7 computer writes a mirror copy of said data on 

8 said second mass storage systCTi. 
9 

0 24. A method for rapid f£Lilure recovery and 

1 system restoration in a fault-tolerant computer 

2 system, the method con^rising the steps of: 

3 (1) obtaining a con^uter system, the 

4 computer system conqprising: 

5 (A) a first server computer system, 

6 con^rising a first computer executing an 

7 operating system; 

e (B) a second server computer system, 

9 con^rising a second computer executing an 
0 operating system; 
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1- (C) a first mass storage system 

2 connected to said first computer; 

3 (D) a second mass storage system; and 

4 (£) means for connecting said second 

5 mass storage system to said first 

6 computer cind to said second computer; 

7 (2) operating said computer system such that 
3 absent a fault, whenever said first computer writes 
9 data to said first mass storage system, said second 

0 computer writes a mirror copy of said data to said 

1 second mass storage system; 

2 (3) detecting a failure of said second 

3 computer; 

A (4) discontinuing causing said writing of 

5 said mirror copy on said second mass storage 

6 system; 

7 (5) remembering data written to said first 

8 mass storage system but not written to said second 

9 mass storage system; 
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(6) configuring said second mass storage 
system to record information from said first 
computer; 

(7) writing said remembered data to said 
second mass storage system; 

(8) \*enever new data is written to said 
first mass storage system, writing a mirror copy of 
said new data to said second mass storage system; 

(9) detecting said second computer's 
availability; 

(10) reconfiguring said second mass storage 
system to record information from said second' 
computer; 

(11) reestablishing data mirroring such that 
whenever said first con^uter writes data to said 
first mass storage system, said second con^uter 
writes a miarror copy of said data on said second 
mass storage system. 
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